Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte Carlo Method

The actor-critic RL is widely used in various robotic control tasks. By viewing the from perspective of variational inference (VI), policy network trained to obtain approximate posterior actions given optimality criteria. However, practice, may yield suboptimal estimates due amortization gap and insufficient exploration. In this work, inspired by previous use Hamiltonian Monte Carlo (HMC) VI, we propose integrate with HMC, which termed as Policy. As such evolve base according our proposed method has many benefits. First, HMC can improve distribution better hence reduce gap. Second, also guide exploration more regions action spaces higher Q values, enhancing efficiency. Further, instead directly applying into RL, a new leapfrog operator simulate dynamics. Finally, safe problems, find that not only achieved return, but safety constraint violations discarding potentially unsafe actions. With comprehensive empirical experiments on continuous baselines, including MuJoCo PyBullet Roboschool, show approach data-efficient easy-to-implement improvement over methods.